Phone Fall Detection and Animation

Leul Tesfaye
05/13/2020

Demonstration Video

Introduction

In this project, I programmed and created a multi-threaded mobile application that dynamically updates UI, establishes a bluetooth pairing with a Raspberry Pi and detect the current free fall state of the phone. While I only physically constructed one Pi attached to a Raspberry Pi 7” touch screen display, this software can accommodate a wide array of display options of Raspberry Pi models. The code running on Raspberry Pi creates two processes that run on two cores. One process serves to receive the data being sent by the phone in near real time while the other process updates the display at its own pace. Overall, the system works well and is actually quite interesting to see the type of plots you can create.

Project Objective:

Develop an app that can detect free fall,compute fall distance,transmit to Raspberry Pi
Establish a stable bluetooth communication
Display data on 7" Touchscreen

Design

I started out by first designing and creating the android application. After reading through the few options for creating mobile applications I opted to use Android Studio due to the large community and readily available documentation.

I experimented with various application software designs ranging from trying to run everything on a single thread to multi-threading each sensor reading,UI updating and bluetooth pairing and transmission. In the end I opted for a 3 thread application. One thread which is responsible for displaying live data, One for establishing and sending data, one for reading motion sensor values and calculating the current state of the phone and current fall height.

After settling on the logistics of the back end. I ended up creating a visually simplistic UI which can be seen above. The UI consists of the toggle button which swaps the Debug outputs between Live update mode and Pause, while the Bluetooth status bar above displays the app's connection status. Once a successful pairing has been made the text will update to “Connected!!!”.

Initially, I planned on computing the fall state using a Simple Threshold and Hidden Markov Model, but I discovered that modern androids actually had a linear acceleration sensor which gives the user X,Y,Z acceleration (without the gravity). Using this I simply computed the difference between this value and the acceleration reading which gave me the current gravity acting on the phone. After saving and analyzing the value I noticed that the deference got close to zero whenever the phone was in free fall. Taking that into account I created my fall algorithm as follows.

                  
                    public Boolean is_falling(){
                        //returns if the phone is currently falling or not
                        float diff[3];
                        for( int i=0; i < 2;i++)     
                            diff [i+1]= Acceleration [i+1] - Linear_Acceleration[i];

                        float temp = diff[0]*diff[0]+diff[1]*diff[1]+diff[2]*diff[2];
                        
                        if (currently_falling){
                            //used to end the timer for a fall
                            if (temp>2.0){
                                currently_falling = false;
                                fall_end_time = System.currentTimeMillis();
                            }
                        }
                        else{
                            if(temp<2.0){
                                if (!currently_falling){
                                    initial_free_fall_vy = vy;
                                }
                                currently_falling = true;
                                fall_start_time = System.currentTimeMillis();
                
                            }
                        }
                        return currently_falling;
                  }

Later in the code, I was able to use the save time and intial free fall velocity values to calculate maximum height by treating this as a simple projectile motion problem. Since velocity was only an approximation and acceleration was prone to random noise. I added a software lowpass filter on acceleration value before computing velocity using numerical integration. This resulted in a very stable and accurate velocity value.

Having the app calculations and transmission sorted out, I began experimenting with various python bluetooth libraries to see which one can create a stable bluetooth pairing. Initially, I started working with Pybluez, a relatively simple to use open source library. While working with this library I noticed that my connection would time out randomly. After spending hours debugging and researching this, I narrowed down the issue to 3 possible causes.

Modern Android RFCOMM connection is unsuited for raspberry pi 3 module.
If the transmission rate from the Android application thread is done too quickly the python application will crash!
If the bluetooth socket is not shut down properly. Meaning the user exits the program using “CTRL + C” or “CTRL + Z”). This will slow down all read executions and eventually cause a time out.

Since there was nothing I could do about the first possible cause, I opted to explore other bluetooth server libraries that had more modern implementations and ended up picking bluedot due to its Non-blocking callback based implementation. I also slowed down the rate of transmission from 90Hz which had it set to in the application down to 30 Hz. Finally, I added a resetting sequence which establishes a connection and closes the server socket 5 times. Note that: these fixes did not completely solve the time out issue; they served to reduce the number of times it happens to maybe once every 20 tests.

Once I had a consistently stable pairing and reading script working. I started to merge in my UI component. Initially, I wanted to use Pyglet, a 3D visualization library that allows for quick and 3D animations. I created a gimbal visualization that can be seen below for pitch and roll readings.

After creating the GUI seen above, I parsed the data being transmitted from the app and displayed it on the GUI. It was during this phase that I started to realize doing a real time update using a 3D library was not going to be possible because of how long it takes to update the display. Parsing the user data and updating each frame takes around half second. Also although Pyglet allowed for 3D visualization of the current readings, it does not contain any functionality for looking how this changed over time.

After spending a good amount of time trying to create a 2D plot embedding in Pyglet, I decided that simply re-write my UI code in matplotlib provided a better user functionality and visualizations capabilities. I created a subplot for pitch, roll and position. I wrote my plotter code such that it dynamically resizes the scaling and animates the fall estimated trajectory as scene in the video. An example test plot can be seen below.

Once I was satisfied with the look and functionality of the python UI, I worked on merging it with the bluetooth server code. Initially I had everything running in the same thread but I soon realized that this resulted in the same problem as the Pyglet implementation. In order to prevent a timeout, I created a multi-process implementation which has the bluetooth server running in a background process which queues all the received data for the foreground process to read and display with about a minute buffering lag. Note it is possible to avoid this buffering lag if one simply disregards any data that isn’t related to the phone falling but the issue with this is the user wouldn’t be able to see the pitch and roll values update on the screen.

Testing

Due to the unstable nature of bluetooth pairing, I had to create several testing scripts to figure out what was failing. I have a list of all the testing scripts that can be found in the Github repo listed below.

Print_data.py (prints the received data from the application)
Cleaner.py (serves to reset the user connection)
Test_connection.py (prints the received app data from within a process)

Besides testing the functionality of bluetooth. I also wrote a script to display the max fall height whenever one was recorded. I tested the accuracy of the fall detection by running this script and dropping my phone repeatedly over measured distances. With the sampling resolution set to 30 Hz my height accuracy of was with in ± 4cm.

Results

Overall, my project worked well. The user interfaces visualizes the data clearly and the touchscreen interaction enables intuitive user movement.The application is pretty stable and easy to use.

There are still some small bugs in the execution. For one, sometimes the application fails to establish a connection and needs to be closed and reopened. Another prominent issue with bluetooth being a lossy transmission system, and some of the data received is corrupted this left out of the animation sequence. I noticed the rate of loss in the transmitted data was directly related to the distance I had the phone away from the PI which makes it difficult to get clear animation data for large falls.

Even with these couple of bugs, the actual visualization is functional and enjoyable to watch. It would just take a while longer to fix these issues to make it a flawless open source project and application.

Future Work

If I had more time to work on this project, I would change the following things. Instead of using bluetooth as means of communication, I would have opted to use online servers to post and get requests. Simply getting a stable and functional bluetooth communication between modern android smartphones and Pi 3 took about 80% of the time in this project. Additionally, I would create my own UI GUI from scratch and use a library like plotly which supports interactive live data updating. From the android application side, I would also add a 3D plot display option so I can compare the actual ground truth data to the losse transmitted information being displayed on the PI.I also would have liked to spend more time trying an offline 3D phone animation where it shows an LG G6 Phone being tossed around in the recorded trajectory. Lastly, I would have liked to explore how to make the app launch based on the drop condition. This way the application can always record the user phone drop and in the case where the phone breaks we have the data to visualize what part of the phone took the impact force.